F5 Guardian Unity Partner
Home » Resources » How the BIG-IP VE system enforces the licensed throughput rate

Topic

The purpose of this article is to explain how the licensed throughput rate is enforced on a BIG-IP Virtual Edition (VE) system.

Description

The BIG-IP VE product license determines the maximum allowed throughput rate. When calculating throughput, the BIG-IP VE system accounts for packets ingressing and egressing the system separately. Additionally, the licensed throughput rate for ingress and egress is enforced separately. For example, if you have a 200 Mbps license, ingress into the Traffic Management Microkernel (TMM) has a limit of 200 Mbps and egress from TMM also has a limit of 200 Mbps.

Ingress packets to be processed by the system are queued in a buffer that is polled frequently. The traffic shaper admits ingress traffic to the rest of the system at the licensed rate from this buffer. This is sometimes enough to shape the traffic with no further action. If bursts of traffic arrive, the buffer begins to fill, and at a threshold some packets begin to be dropped to signal to sending TCP stacks that congestion is occurring. Normally drops are very light and TCP retransmissions replace them. Light drops are a normal part of the traffic shaper mechanism and are more common with lower-licensed rates and higher numbers of TMM threads. Drops are tied to buffer usage and not to any measurement or logging of ingress traffic in a period, although there is a clear relationship between the two:

For ingress, the system is very similar in behavior to a network switch, accepting traffic over a high-bandwidth link and passing it on through a lower-bandwidth bottleneck link.

Throughput for egress traffic is determined by the number of bytes sent over predefined, sliding time windows. If the number of bytes sent surpasses the limit for a given time window, the rate shaper starts limiting the egress traffic.

The rate shaper throttles throughput to the licensed rate at the packet level, rather than at the connection level. Ingress and egress packets are dropped during the limiting process and no management is done at the connection level.

Important: For BIG-IP versions prior to 12.1.0, promiscuous traffic received on the BIG-IP interface is considered when determining the throughput rate. Starting with BIG-IP 12.1.0 and later, the throughput calculation no longer includes promiscuous traffic (for example: for BIG-IP 12.1.0 and later, failover, heartbeat, mirroring, configsync, and monitors are no longer counted as throughput traffic. Regular BIG-IP traffic is still counted (including ARP, broadcasts, UDP, and TCP).

Note: The licensed bandwidth is dynamically distributed among TMM threads several times a second. This allows for a single connection to consume most of the bandwidth on an otherwise-quiet BIG-IP VE while still reasonably distributing licensed bandwidth during typically busy periods. Before BIG-IP version 11.6.1, each TMM thread has a fixed equal share of the bandwidth.

Beginning in BIG-IP 11.4.0, when the throughput rate exceeds 75 percent of the maximum allowed throughput rate licensed, a notification similar to the following example is logged in the /var/log/ltm file:

01010045:5: Bandwidth utilization is 8 Mbps, exceeded 75% of Licensed 10 Mbps

Note: Bandwidth utilization measurements and logging are based on 0.5 second intervals. You should judge the log messages collectively and not individually. You can better determine whether or not the BIG-IP VE is congested or busy over meaningful time periods by looking at throughput graphs that average over longer time periods: 10 seconds for three-hour graphs, 30 seconds for 24-hour graphs, one minute for seven-day graphs, and 10 minutes for 30-day graphs. Bursts of traffic may register in the logs but appear increasingly lower in graphs because they are averaged over increasingly longer periods as the data ages. It’s also normal for bandwidth to reach the licensed limit for larger data transfers (relative to the bandwidth), particularly if the licensed limit is a bottleneck bandwidth, and be shaped to that limit with few drops as long as the BIG-IP VE system is not congested.

Recommendations

Determine licensed throughput

To determine the maximum allowed throughput rate for a BIG-IP VE system, perform the following procedure:

Impact of procedure: Performing the following procedure should not have a negative impact on your system.

  1. Log in to the Traffic Management Shell (tmsh) by typing the following command:
    tmsh
  2. To display the maximum allowed throughput rate, type the following command:
    show /sys license detail | grep perf_VE_throughput_Mbps

View dropped ingress/egress packets

To view the number of ingress or egress packets that have been dropped, perform the following procedure:

Impact of procedure: Performing the following procedure should not have a negative impact on your system.

  1. Log in to the BIG-IP command line.
  2. To list the number of ingress/egress packets that have been dropped by each TMM, type the following command:
    tmctl -d blade tmm/if_shaper

The output of the command appears similar to the following example:

Note: The following output is from an idle system that has not experienced any ingress or egress packet drops.

# tmctl -i -d blade tmm/if_shaper -w 180

shaper_tid ingress_max ingress_mbo ingress_red ingress_drops ingress_shaped_bytes ingress_unshaped_bytes egress_shaped_bytes egress_unshaped_bytes egress_mbo egress_drops

shaper_tid ingress_max ingress_red ingress_drops egress_drops
1 0 0 0 0
0 0 0 0 0

Note: It’s typically most useful to calculate a packet drop rate between two samples. This example doesn’t record packets through the system (though the tmm_stat table does), but a quick way to approximate packet drop is to divide ingress_shaped_bytes by 1500 to get a probably low estimate of packets that have passed through the shaper. You can express the ingress_drops divided by this packet number as a percentage. Usually, drops greater than about 1% are considered high, although this is an approximation not suited for every situation. Unshaped traffic, such as performing a ConfigSync, does not count against your license.

Note: A correctly sized system may have a non-zero number of drops. A light rate of packet drops is a normal part of traffic shaper operation and is not detrimental to TCP traffic. Heavy dropping, particularly coupled with obviously high use in throughput graphs, likely indicates need for increased bandwidth. It’s not typical, but, for example, if traffic arrives in a substantially synchronized way every minute at the top of the minute, then you may need to over-provision bandwidths to meet requirements for latency and drops at these times.

Supplemental Information